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ABSTPACT 

The training package evaluated in this report vas 
developed to train school administrators and graduate students in 
educational administration in school program planning skills. The 
package consists of a coordinator handbook and units on setting 
goals* analysing problems and deriving objectives. Each unit is 
coBposed of four«to*six modules that provide training on a limited 
number of instructional objectives. A typical module contains reading 
material* individual or ssall'-group activities in vhich the trainees 
practice the skills* and feedback for the practice activities. 
Evaluation inforiaation vas collected in three areast (1) attitudes 
toward the units and attainment of their instructional objectives by 
persons who are trained with the units* <2) use of the units in both 
schools and universities* and <3) evaluation of the units by 
knowledgeable reviewers. (Author/MLP) 
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1. INTRODUCTION 



The pctotmjnlng Instnictlnnal Purposes training package was developed 
by the Far West laboratory for Educational Research and Development (FWL) 
for the purpose of training school administrators and graduate students 
in educational administration in skills related to planning of effec- 
tive school pr jgranis. A contract for an external evaluation of the 
training package, tc be conducted between May, 1974, and December, 1974, 
was awarded by FWL to Instructional Technology Consultants (ITC) of 
Scottsdale, Arizona. This document was prepared by ITC as the final 
report to FWl. on the external evaluation of the training package. 

Descripti on of the Training Pack a ge 

Th*> nptpninninn Tn<;trurtinnal Purnosps traininn nnrkagp rnn<;i«;t<i nt 
a Coordinator's Handbook and three training units: Unit 1— Setti ng Goals, 
Unit 2- -Anal yzing Prob iems, and Unit 3— Deriving Objectives . Each unit 
is comprised of four-to-six modules which provide training on a limited 
number of instructional objectives. The module names and the objectives 
for each module are contained in Exhibit A of the exhibits submitted as a 
separate document accompanying this report. A typical module contains 
reading material related to the skills taught in the module, individual 
or small group activities in which the trainees practice the skills, and 
feedback for the practice activities. In many of the practice activities, 
the trainees are organized into planning teams in order to apply the skills 
covered in the module to problems and proccesses in a hypothetical school 
district. 
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The units wore desiyned so that either one unit only or any tombinu- 
tion of the three units could be used with a group of trainees. Thus, 
if the administration in a school district wished to concentrate their 
current planning efforts on only one of the three general processes 
(setting goals, analyzing problems, or deriving objectives) covered by the 
units, they could use only the unit dealing with their particular area 
of concern. The units are intended to be self-contained in the sense 
that step-by-step progress through the materials and activities coin- 
prising a unit should result in attainment of the desired outcomes by 
participants. A unit may be administered eUher in a concentrated short- 
term workshop or in individual sessions scheduled over a period of 
several days or weeks. FWL's estimate of the training time required for 
the units is 10-15 hours each for Units 1 and 3 and 12-18 hours for Unit 2. 

ine training materials stipulate that workshops or other types of 
training courses conducted with the materials should be directed by a 
coordinator. According to the program developers, the coordinator should 
have worked through the relevant unit(s) on his own or as a member of 
a training group, but no other prior knowledge of the content area is 
required. It is not necessary for the coordinator to provide new sub- 
stantive infonriation during a training course. Instead, the coordinator's 
role involves organizing, guiding, and monitoring activities in which 
participants use the materials and procedures contained in the units. 
Detailed procedures and other relevant information for coordinating a 
training course are provided in the Coordinator's Handbook which is a 
part of the training package. 

The Coordinator's Handbook and all materials in the three units are 

entirely in print form. The units range in length from 155 pages to 
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259 pages. Ihey are available from FWL at a cost of $8.95 per single unit 
or $?4.95 for a set of the three units. The Coordinator's Handbook, 
which contains the directions for coordinating all three units. Is priced 
at $4.50 per cop>. 

Evaluation Procedures 

The evaluation procedures employed by ITC were designed primarily to 
yield infomjation that would provide a comprehensive and accurate summa- 
tive evaluation of the training units and that would be helpful in making 
decisions about their dissemination and use. Information was collected 
In three areas: (1) attitudes toward the units and attainment of their 
instructional objectives by persons who are trained with the units accord- 
ing to the procedures described in the Coordinator's Handbook, (2) use 
of the units in both schools and universities, and (3) evaluation of the 
units by reviewers who possess recognized expertise in the content areas 
covered by the training materials. 

Collection of Information In the three areas listed above was accom- 
plished through a variety of evaluation activities, to collect data on 
attainment of the instructional objectives of the units by trainees and 
on trainee attitudes toward the units, a concentrated four-day workshop 
was conducted in which the units were used to train a group composed of 
school administrators and university faculty members. Information on use 
of the units was collected through mailing of a brief questionnaire to all 
workshop participants and all purchasers of the units, telephone inter- 
views with persons who had used the units, visits to two school districts 
In which the units had been used, and obtaining of reports on the use of 
the units at three additional sites. Comprehensive reviews of the units 



were obUlmul from two university faculty nieinbers who have done consider- 
able substantive work in the content areas covered by the units, and 
a third review was obtained from a district-level school administrator 
who was very familiar with the units and who had served as a coordinator 
in the workshop conducted as part of the external evaluation. 

Presented In the following three sections of this report are descrip- 
tions of the workshop based on the training units, the use of the units 
by purchasers and workshop participants, and the reviev/s of the units. 



2. THr PLTMI1IN6 .jN5^^^^^^ WORKSHOP 

A four- day workshop in which the units were used to train a group of 
school administrators was held from August 12 through August 15, 1974, 
at the Airport Marina Hotel in Burlingainc, California. Arrangements were 
niade by ITC to hold the workshop under the sponsorship of the Association 
of Califon.id School Administrators (ACSA), the major professional 
organization of school administrators in the state of California. The 

« 

primary purposes of the workshop were to obtain Information related to 
(1) the effects of the training on participant attainment of the Instruc- 
tional objectives of the units and (2) the attitudes of participants 
toward the units. 

To publicize the workshop, approximately 2350 copies of a pamphlet 

hi fhf» i.in*»l'c hnn Ar»H rnnt»Sn<nn an anr>Hrat<nn fo**}n for W****? 

mailed to selected agencies and Individuals. Included in the mailing 
were all school districts in the Unites States with pupil enrollments 
of 10,000 or higher, all school districts in California, the educational 
administration departments in all universities In the United States 
with enrollments of 10,000 or higher, the major state education agency 
in all 50 states, and the Individual participants in ACSA's Project 
Leadership. A letter of acceptance was sent to each person who submitted 
an application form for the workshop 

Works hop Parti c ipjnts and Staff 

A total of 25 participants attended the workshop. Twenty- two of the 
participants were from school districts in California. The remaining 
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three participants were faculty inoi:ibers from universities In Michigan 
and Utuh. A list of the participants and staff Is contained in Exhibit 
B In the exhibits submitted with this report. 

Two district-levol school administrators--Mr. Jamo? Freda, Director 
of tvaludtion in Parumount (California) Unified School District; and 
(Jr. James Zaharis, Assistant Superintendent of Schools in Mesa, Arizona- 
wore employed by IIC to servo as coordinators during the workshop. All 
instructional activities during the workshop wore directed by the. 
coordinators. Prio.' to the workshop, each coordinator worked through 
the Coordinator's Handbook and the un1t(s) for which he was responsible 
during the v;orkshop. 

While the coordinators were the key staff for the workshop, two other 
staff r.crbers played lesser roles. Dr. Edward Besubier, Assistant Lxecu- 
tivo of Af.sA, and Or. Howard Sullivan of ITC welcomed the parti- 

cipants at the opening session of the workshop. Or. Sullivan also 
administered evaluation instruments to participants on several occasions 
during the v/orkshop. 



Organization a nd Schedule 



The workshop was organized according to the following schedule: 



Monday to Wednesday Morning: Setting Goals (Unit 1) 

"Dr. 7"ahariVr Coordinator 



Analy zing Problems (Unit 2) 
Tir. Freda, Coordinator 



Wodnesdjy Morning through 




Dr". Za'ha r is ,'"Coord i na tor 



At-'>'iving Object 1 VPS (Unit 3) 
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Offorifuj units on tho schedule llbU'd above enabled each parti- 
cipant to complete I'ny two units of his choice from eniomj tho three units 
coHipris iiKj the t^Mminlrjcj Jniit^^^^^ training packayo. 

The sch'dult? of units for the workshop was explained to participants 
durinu the openinQ session on Monday niornimj. Ana^l^zij»(iJ|rjDh1oii^ (Unit 
2) was offered twice because It was anticipated that tho laryost nuiiil)er 
of p.uticipjr»ti> would select it and that reaction tov.'ard It would bo 
most fav(jiv.ble. The number of participants sclectinc) each unit wa:> as 
folUn.'s; 

St»ttin<: no.il:; (M-W) 10 Analyzing Probloins (W-Th) 4 

A«L^«Jy/li'i!.!!v'ct)li";s (M-W) 15 El^yTnOWckl^ (W-Th) 21 

Because the workshop was designed as a means of evaluating the units, 
it was. corulucted so that participants worked through the Instructional 
nt.i»» »-i,4l«: .111(1 «u tiwi tif>«; (l».'«;rri in thr» i»nit<i flnH in tho ToorHinntor ' 
MandL» ok. The coordinators were directed to eschew the type of choosing 
aiiiong n'odulcs and/or activities that might occur in many instructional 
settings and to complete the units In a manner that would permit evalua- 
tion of their effects when they are used as prescribed by the developer. 

Assessment^ Inst m^^^ and Procedures 

Two types of assessment instruments were developed for the purpose 
of collecting evaluation data from workshop participants. Unit tests 
keyed to the instructional objectives of each unit served as measures 
of parti 'J pant attcinment of tho unit objectives. A unit rating form 
was d^^ivjMfd as an indicator of participant attitudes toward each unit. 

The unit tests were intentionally constructed to be short tests 
in ord.T to minimize the amount of ti..'.e that U would be necessary to 



duvolf to te<,t1rH) durinj the worksho}). Two tost fonic cuiiloininu i»n 
equal mv.hcr of itLiia> on odth form wrv dcvolopinl ^or each unit. All 
Items wore multiple- choke items with either four or five response 
thoict^. per item. There wore a total of six iteius per form on the two 
forms for tl>e Srttin<] Goalr. Dorivjn^i Objcctivcr. units. Tlic two 
foivr. for Afuily.'ir.cj P rub I » which is the lonfjtst of ihc three units 
and hus the must ir.structiohcil objectives, contained 11 items each. 

All items for the unit tests were initially written and asslcined 
to test forms by Lducational tvuluation Associates of Los Angeles under 
an aciroemcnt with ITC. Many of the items were subsequently revised by 
ITC on the basis of reviews by FWL and ITC of the initial set of items. 
A description of the procedures involved in developing and keying the 
test items* assiynini) items to forms of the test, and trying out the 
j^r- H- pv-ir»>- to the workshop is rontainon in Mppeiiuix A. Al^u prebtnUu 
in Api)endix A us an exiimple of a unit test is Form A of the unit test 
for Setting Goals. Copies of both test forms for each of the three units, 
with each item referenced to the module and objective for which it was 
written, are contained in Exhibit C. 

Participant attitudes toward each unit were assessed with a Unit 
Rating Form developed by ITC. This form contained 20 statements— 10 
positive and 10 negative— about the units. Each participant in a unit 
indicated his degree of agreement with each statement on a five-point 
Likert-typc scale. The same 20-item fortn was used for rating each of 
the thrt'C units. Participants were also ac-ked to make suggestions for 
improvement in the unit on this form and to list any additional consents 
that they wished to make about the unit. 



All participants a particular unit were pretested on the unit 
iiwiied lately prior to beginning work on it. Forms A and B of the unit 
test were each assigned at random as the pretest to half of the parti- 
cipants in the unit. Following completion of the entire unit, the Unit 
Rating Form was adwilnistercd to all participants. Participants completed 
this fonn and then took the unit posttest. Participants who had taken 
Form A of the unit test aj. the pretest took Form 8 as the posttert, and 
those who were prt;testod with Form B took Form A as the posttest. 

The 10 participants in Setting Goals and the 19 participants In 
Aj>.a.lyzing Problpiiis were present for all training sessions with these 
units an. I completed both the pretest and posttest. Nineteen of the 21 
participants In Derivi ng Ob jectives were present for all activities 
related to this unit, Including both testing sessions. The other two 
p<»» lit i^»allt,i ill Liie uiii t were presient on tne rmal oay or the workshop 
but were excused from the posttest because they had been absent from the 
workshop on school -related business during some of the unit activities. 

Results 

Mean scores on the pretest and posttest for each unit are shown In 
Table 1 on Page 10. As shown in the table, for the Setting Goals unit 
the pretest mean score was 1.9 items correct (32^0 and the posttest mean 
was 4.5 items correct (75%). Mean scores for the A nalyzing Pro blems 
unit were 4.5 {A]%) on the pretest and 6.7 [61%) on the posttest. For 
DpIJJ/Jilil.. P>J PC ti^^^ the pretest and posttest means were 2.2 (37%) and 
2.8 (48^ respectively. 

Participant responses to the 20 statements on the Unit Rating Form 
are sunnarized for each unit In Table 2 on Page 11. (A copy of the Unit 



Table 1 

PRETEST AND TOTTEST MIAN SCORES BY UNIT 



No. of Pretest Posttest 
Unit N Items Mean Mean 



1. Setting Goals 10 6 1.9 4.5 

(32%) (75%) 

2. Analyzing Probleins 19 11 4.5 6.7 

(41%) (61%) 

3. Uerivinq Objectives 19 6 2.2 2.9 

(37%) (48%) 
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Tabic 2 
SW^IARY OK UNIT RA1JNGS 



Ratings b y U nit on Pos i tive $tator.»onts 



Item 


Set tint) 

virt 


Coals 
4 ' 5 


Analyzln^^ 


Problems 

■4 5' ■ 


Deriving Objocti 
•""T T""r'"4-"F 


2. 


*3 


6 


0 


1 


0 


6 9 


4 


0 


0 


6 14 


0 0 0 


4. 


3 


7 


0 


0 


0 


7 11 


0 


0 


1 


9 11 


0 0 0 


5. 


1 


8 


1 


0 


0 


3 12 


3 


1 


0 


4 14 


2 0 0 


6. 


2 


b 


3 


0 


0 


2 6 


5 


5 


1 


2 13 


1 4 0 


8. 


0 


8 


1 


0 


0 


2 6 


7 


4 


0 


2 12 


3 2 1 


10. 


0 


7 


1 


1 


1 


2 9 


4 


4 


0 


5 8 


4 2 1 


13. 


1 


6 


2 


1 


0 


3 7 


4 


4 


1 


5 8 


3 3 1 


15. 


2 


7 


1 


0 


0 


3 11 


3 


2 


0 


5 14 


0 0 0 


16. 


0 


3 


4 


3 


0 


2 9 


3 


2 


3 


2 11 


3 3 0 


18. 


1 


4 


3 


2 


0 


4 9 


3 


3 


0 


1 11 


3 3 1 



Total for 



tacn Kttiincj 


13 61 


16 


9 


1 


34 89 


36 25 


6 


41 11619 


17 4 


% Choosing 
























Each netimi 


13 61 


16 


9 


1 


18 A7 19 13 


3 


21 59 


10 


9 2 














Ratings by Unit on Kefit'tlye Stateipents_ _ 






Item 


Setting Goals 


Analyzing Problems 


Deriving Objectives 


NuFii)«r 


1 


2 


3 


4 


5 


1 2 


3 4 


5 


1 2 


3 


4 5 


1. 


0 


3 


2 


5 


0 


2 2 


6 7 


2 


0 5 


4 


8 3 


3. 


0 


1 


3 


6 


0 


2 2 


1 11 


3 


0 2 


3 


13 2 


7. 


1 


1 


3 


5 


0 


8 6 


2 0 


3 


3 3 


4 


6 4 


9. 


0 


2 


1 


7 


0 


0 2 


4 10 


3 


1 4 


4 


10 1 


11. 


0 


1 


2 


5 


2 


2 2 


1 10 


4 


0 2 


5 


8 5 


12. 


0 


0 


2 


6 


2 


1 4 


7 4 


3 


0 3 


1 


12 4 


14. 


0 


7 


1 


1 


1 


3 6 


0 9 


1 


2 4 


4 


8 1 


17. 


0 


5 


0 


5 


0 


2 7 


1 6 


3 


1 9 


4 


4 1 


19. 


0 


1 


4 


5 


0 


0 2 


6 7 


3 


0 2 


7 


5 4 


20. 


1 


4 


3 


2 


0 


4 9 


3 3 


0 


1 5 


6 


4 3 


Total for 






















78 28 


Each Rating 


2 25 21 


47 


5 


24 42 


31 67 25 


8 39 


42 


% Choosing 




















2? 


40 14 


Each Ratlr.g 


c 


?5 


21 


47 


5 


13 27. 


16 35 


13 


4 ?3 



♦The entries In the row for each Item show the number of persons \/ho chose each 
rating from 1 (strong agrecr.ient with staterient) to 5 (strong disagreement). Cn 
item 2, for exanple, three participants in the Setti ng Goa ls unit marked Choice 
1, six marked Choice 2, etc. Complete Unit Rating Forms containing all 20 state 
ments are Included In Appendix B. 



Ratint} Forin for each unit, containing the 20 statements on the form and 
a suninary of the responsos to e".ch statement, are presented In Appendix 
B on Payes 55-60. Responses to the 10 positive statements about the units 
are reported S(!parate1y in Table 2 from responses to the 10 negative state- 
ments. On each of the 10 positive statements, a rating of 1 is the most 
favorable rotiny (strong agreement with the positive statement) and a 
rating of 5 (strong ('isagrocment with the statement) is the least favor- 
able rating. On the 10 negative stf.tements, a rating of 5 is most favor- 
able and a rating of 1 is least favorable. 

Ihe overall reaction to the units can be seen by examining the per- 
centage of participants who chose each rating, as shown in Table 2 on 
the bottor. line for both the positive statenents and the negative 
stateixTts. For the 10 positive sta*wcn.ents , the most frequently chosen 
rating for e.^rh of the 3 units wes Choice 2, Indicating agreement with the 
positive statements about the unit. For the 10 negative statements, the 
nost frequently chosen rating for each of the 3 units was Choice 4, Indi- 
cating disagrecnx?nt with the negative statements about the unit. The 
data on percentage of participants who chose each rating also reveal 
thct the total nuntor of favorable ratings (Choices 1 and 2 on the positive 
statements and Choices 4 and 5 on the negative statements) for each unit far 
exceeded the total number of unfavorable ratings for the unit. 

The individual items which were considered to be particularly good 
indicators of a participant's overall reaction to a unit were Items 11, 
12 and 13. These three items deal with whether the participant plans to 
use the unit in his school or district, his rating of the unit compared 
to other administrative training he has received, and whether he would 



d<Jvi«ie others to attend a workshop based on the unit. The data in Table 
2 reveal favorable ratings on these three items for all three units, 
except for an overall rating in the neutral range for Ajnajy zing £rpb^^ 
as coiupared to other administrative training (Item 12). 

I'xami nation of the ratings in Table 2 on an item-by- item basis also 
reveals the characteristics of the units about which participants held 
the most favorable attitudes and the characteristics toward which their 
attitudes were least favorable. Ratings across all three units were co?!- 
sistently high on the potential usefulness of the unit for training school 
administrators (Item 2), the importance of the skills taught in the unit 
(Item 4), the usefulness of the information and practice provided in the 
instructional activities (Item 5), and the appropriateness of the unit 
for use in a graduate course in educational administration (Item 15). 
Ratings across the three units were lowest on Items 17 and 20, indicating 
that the participants need more training in order to serve as a coordina- 
tor fur the unit (Item 17) and that the units require too much time to 
be used in a school or district during the school year (Item 20). 
Participants in the Setting Go als unit indicated that their unit would 
be difficult to use for the purpose of training others (Item 14), and 
participants in Analyz ing Problems indicated agreement with the statement 
that the skills taught in their unit could be taught in far less time 
than the unit presently takes (Item 7). 

The Unit Rating Form also contained items asking the participants 
to suggest potential improvements in the unit and to make any additional 
conunents that they wished to make. Responses to these items are listed 
for each unit in Exhibit D. The most frequent suggestion was to reduce 
the length of the units and/or the amount of reading required. This 
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ougyustion was iiutde very froquencly for dfjcl And lyzing PrubU tiis unit tind 
sovcral timuo for D o r 1 v i ny Ob j oc t j y os . 

Suntn ary and Discussion 

The DcAorniintn^ Instructional Purposes workshop was conducted as 
part of the external evaluation of the three training units. The prin- 
cipal sources of evaluation data collected during the workshop were 
(1) the pretest ?.nd posttest scores of participants on unit tests cover- 
ing the objectives of each of the training units and (2) participant 
ratings of each unit. Data related to post-workshop use of the units 
by each participant were collected approximately six weeks after the 
workshop and are described in a later section of this report. 

The degree of pretest-to-posttest improvement registered by 
participants on the unit tests varied considerably across the three 
units. Pretest mean scores, which ranged from 32 percent on the Setting 
Goals unit to 41 percent on Analyzi ng Objectives , indicate thai; partici- 
pants could not perform v/ell prior to Instruction on the skills taught In 
the units. Following instruction, the participants perfonned moderately 
well (75 percent) on the posttest for Setting Goals and rather poorly 
(60 percent and 48 percent, respectively) on the posttests for Analyzing 
Problems and Deriving Objectives . Posttest performance on the units 
could be considered to be reasonably satisfactory only on Unit 1, Setting 
Goals . Participants made rather modest pretest-to-posttest gains 
(11 percent and 20 percent) on the other two units, but their posttest 
scores on these units Indicate that, following Instruction, they still 
were unable to perform well on the specific skills taught in the units. 



Ptirticipant r\»tiii(jo of thu unito on thi: Unit Katltu; Form rofloct 
favoro!»le attUudt-s toward each of the thr-.'u units. Positive rdtings 
outnuniberod nuyutive ratinys by a ratio of v.oro than thrco to one on 
Units 1 and 3 and inore than ivio to one on Unit 2. By far the most 
coiunonly sclec:tt;d rating was a ? for positive statcincnts about the 
units aiui a 4 for negative statements. In gcnoral, the ratings indicate 
that participant attitudes toward the units were consistently favorable, 
even Uiough they typically wore not in JiO "strongly favorable" category 
rcprcsenlod by a rating of 1 on positive statements and a rating of 5 
on negative statoincnts. 

The responses to individual items on the rating fonn revealed that 
participants felt that the skills taught in the units were important for 
administrators to learn, that they learned a lot from the units, and 

tOat tiicjr VvOUlU dUuocv.{Ut:ii L ly Uoc bilb uii i UiiU/Oi vvwt Ndiiu^ ktoitiin^ iii 

their school or district. Ratings were also generally favorable on the 
other statements included in the 20-item form. The only ratings which 
were in the neutral -to-negative range across the three units related to 
the amount of time required by a unit for use in a school or district 
during the school year and to the need for additional training before a 
participant could serve as a coordinator for the unit. It seems probable 
that the concern about the need for additional training would be partially 
alleviated by use of the Coordinator's Handboolc, which was not available 
to participants until after they had completed the units and the rating 
form. 

Contrary to pre-workshop expectations, the ratings on Unit 2, 
although favorable, were slightly less positive overall than the ratings 



on Ufiits 1 and 3. One reason for the lower ratirjgs on Unit 2 appears 
to be its greater length. Concern over the atiiount of time required by 
the unit:, was reflected in the ratings of all three units, but It was 
particularly strong in the case of Unit 2. The greatest number of nega- 
tive written coiisiicnts on the Unit Rating Form also related to the length 
of Unit ?. 

Even though the rwrticipants reacted favorably toward the units, it 
seems likely that trainees' attitudes would be more favorable under con- 
ditions that are more typical than those that prevailed during the work- 
shop. In a typical situation the coordinator would be free to select the 
activities and materials that he feels are most useful, rather than hav- 
ing to use the intact units as the coordinators were instructed to do dur- 
ing the workshop. Under such conditions, the characteristic of the units 
th.it drew the most unfavorable reactions from the participants— thpir 
U'iiglh or wordiness--should be of much less consequence. 

As a result of the experience gained in monitoring the workshop 
and evaluating the units, it appears to the evaluator that the set of 
instructional objectives for each unit is not generally useful either as 
a basis for evaluating the unit or for developing its instructional con- 
tent. For evaluation purposes, it seems more appropriate to establish 
the desired outcome(s) for each unit at the level of one or more goals 
indicating the broader skill a person would display and/or the product 
he would develop as a result of completing the unit. For example, the 
goal for Unit 1, Settinja^GoaH^, might be as follows: 

Derive one or i.iore goal statements for a school or district 
and a set of prioritized indicators for each statement. 
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The effectiveness of the urnt would then be evaluated at its conclusion 
primarily on the bnsis of whether the user could develop (or had developed), 
according to the relevant criteria, the product{s) specified in the unit 
goal . 

There are two major reasons for sutujesting the use of one or more 
goals per unit as the primary basis for evaluation of performance on each 
unit, ratiier thnn using the comprehensive sets of unit objectives which 
served as the basis for evaluating performance during the workshop. First, 
a single goal or a small number of goals for each unit appear to ade- 
quately cover the really important outcomes of the unit and to be the 
best indicators of the outcomes that are desired for each unit by users. 
Second, the nature of the materials and their intended use is such that 
users should be able to work with the actual materials throughout the 
procc^o wf u<.;»c'Iup*:..y l!»c uuit Tclatcd product cither in a training 
session or while working with a school or district team. However, success- 
ful performance on many of the instructional objectives requires recall 
of rules, definitions or criteria which are included in the units. There 
does not seem to be a compelling reason to require users to commit such • 
nvTterial to memory. Instead, it seems much more appropriate even during 
training to permit them to use the information in the units as resource 
material and to evaluate them primarily on the basis of the product they 
produce. 

In general, the two major sources of data— participant performance 
on the unit tests and participant ratings of the units— collected during 
the workshop yielded rather divergent evaluation information about the 
units. The ratings of the units revealed that participants held favor- 
able attitudes toward all three units. However, participants did not 



per fori.1 well on two of Ihu three unit posttosts covcrlmj the instruction- 
al object ivoii of the units. The evalua tor's experience with the work- 
shop and units led him to conclude that assessment of performance on 
the comprehensive set of objectives for each unit is not a particularly 
useful basis for evaluating the effects of the units. In the evaluator's 
opinion, the data collected during the workshop yield an overall evaluation 
of the units that is generally, but not extremely, positive. 
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3. USi: OF THE UNITS 

A series of activities was undertaken from September through November, 
1974, by the external evaluator to obtain information related to the use 
of the units by the workshop participants and by persons who had pur- 
chased the units from FWL. Initially, a brief questionnaire was mailed 
to all purchasers of the units other than librarians who were identified 
by name on the list of purchasers maintained by FWL. A similar, but not 
identical, questionnaire was mailed to all workshop participants. Sub- 
sequently, all perions who returned a completed questionnaire and indicated 
on it that they had used one or more units to instruct or train others 
were contacted by telephone regarding their use of the units. Based on 
the telephone interviews, several sites were identified where one or 
more of the units had been used rather extensively. Visits were made to 
two such sites at which the users had been workshop participants, and 
written reports were obtained from individuals who played key roles in 
the use of the units at three other such sites. 

Questionnaire Procedures and Responses 

To obtain initial information on use of the units, questionnaires 
were constructed that were intentionally very brief, easy to complete, 
and easy to return. The major purposes in using the questionnaire were 
to identify persons v/ho had used the units and to obtain an indication 
of their willingness to talk briefly by telephone about their use of 
the units. The questionnaire items for both workshop participants 
and purchasers were printed on one side of a standard-sized postcard, 
which was stamped and addressed on the reverse side for return mail to ITC. 



A brief cover letter and the purchaser postcard -questionnaires were 
njdlled to 172 purchdscrs in mid- September. The questionnaire cards for 
workshop participants and an accompanying cover letter were mailed to all 
25 participants in late Scptcrnbor. Completed questionnaires were received 
from 61 of the 17? purchasers {3S%) and from all 25 of the workshop parti- 
cipant*;. No follow-up mailing was conducted to purchasers who did not 
return the questionnaire card from the initial mailing. 

The questionnaire items on the cards for both workshop participants 
and purchasers and a tabulation of responses to each item are shown in 
Table 3 on Page 21. It can be seen from Item 1 on the participant question- 
naire that 12 of the 25 participants (48%) indicated that they had used 
one or more of the units with others in the period between the August 
workshop and the time that they completed the questionnaire. These 12 
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units were used, since two instances involved use of the units by different 
teams of three participants each and a third case involved use by a team 
of two participants. Seventeen participants (68%) reported on Item 3 
that they planned to use the units at some future time. Examination of 
the responses of each participant to both Item 1 and Item 3 revealed that 
20 of the 25 participants (80:') reported that they either had used one or 
more units to date or planned to do so in the future. Twenty participants 
(80.;) also reported on Item 2 that the content of the units had been use- 
ful to them in some way other than using them directly with others. 

Item 2 on the participant questionnaire reveals that 10 of the 61 
purchasers (16:0 who returned the questionnaire card indicated that they 
had used the units to instruct or train others. Thirty-six purchasers 
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Table 3 

QUESTIONNAIRE RESPONSES OF WORKSHOP PARTICIPANTS AND PURCHASERS 



♦RESPONSES TO PARTICIPANT QUESTIONNAIRE 

1. Since the August workshop, have you used one or more of the Det ermini ng 
I'lStructipxi-y^jrj^jlliJoses units with others? 

lu yes J£ no 

2. Has the content of the units been useful to you in any other way? 

y.O yes £ no 2_ (no rceponuc) 

3. Do you plan to use one or more of the units in your school or district at 
some future time? 

17 yes no "uncertain" 

4. We would like to telephone several of the workshop participants and to talk 
briefly with them about the units. Would you be willing to talk with us by 
telephone for a very few minutes? 

22 yes 2 no 3. (no veeponse) 

*A11 25 workshop participants returned completed questionnaires. The number 
beside each response choice indicates the number of respondents making each choice. 



♦ ♦RESPONSES TO PURCHASER QUESTIONNAIRE 

1. Have you reviewed one or nore of the Determining Instructional Purposes units? 

Bl yes 8_ no 2 (no responee) 

2. Have you or a colleague of yours used one or more of the units to instruct 
or train others? 

10 yes 48 no 3 (no response) 

If yes, please describe the context in which you used the unit(s). 
If no, please state why you have not used them. 

(Written rct^pcuoco to this item are deooribed in the text of the report*) 

3. Do you plan to use the units at some future time? 

36 yes 14 no 5 "uncertain" 3 (no response) 

4. (Answer only if you have used one or more un.ts.) 

If you have used one or more units, we would like to talk with you briefly 
about them. Would you be willing to talk with us by phone for a very few 
minutes? 

yes 2. no 1_ "not ford liar enough with them" 

**Sixty-one of 172 purchasers (35%) to whom questionnaires were mailed completed 
and returned them. 
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:) rcpoi ted in 3 that tlity planned to ui.0 the umts at some; future 
tiiro. ami ojyht otliur'^ (13.) wrote brief coi.iiiionts stating thcit they were 
uncort.iin \nth reyard to future use of the units. Fifty- one of the 61 
purthai,er.i ) indicated that tlioy had reviewed one or more of the units. 

Of tlio 48 persons who reported that they hod not yet used the units, 
30 did not »obpotui to the part of Item ?. in which they were asked to state 
why thty had not done so. Four of the 48 respondents X^t) who had not 
used the units stated t-.at the reason was either weaknesses in the materials 
or that they were too tinie-consuinlng. No other reasons that were cited 
for not using the units were negative in nature. 

In the follow-up telephone interviews, each of the purchasers who 
had used ti. • units was asked to describe the context in which the units 
w(»rp used. Since more detailed descriptions of the context were obtained 
Juiir.r, Xc^c]^\lr^v conversationn than on Itw ? nf the questionnaire, 
this information is reported below in the section describing the telephone 
interviews . 

In all, a total of 22 individuals-12 workshop participants and 10 
purchasers— reported that they had used the units. Thirteen of the users 
were school administrators in a local education agency, seven were college 
or university faculty members, and two were staff members of state educa- 
tion agencies. Nine of the 12 users from the workshop were from California, 
two from Utah, and one from Michigan. Of the 10 purchasers who had used 
the units, two were from Florida, two were from Rhode Island, and one 
each wc\s from California, Hawaii, Idaho, Kansas, New York and Texas. 
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Data related to use of the units wnro obtained through telephone 
inttrvicws with selected Individuals who had completed and returned the 
questionnaires. The interviews with persons who reported that they had 
used one or niore units provided information on nearly all of the situa- 
tions indicated on the questionnaires in which the units had been used. 
Interviews were also held with a limited number of persons who reported 
that they had reviewed the units but had not used them. 
Interviews with Users Telephone Interviews were held both with work- 
shop participants and with purchasers who reported on. the q.uestionna1res 
that they had used one or more of the units. A telephone Interview was 
held with ore workshop participant at each of the seven sites at which 
tho \tn\f*i h,iH hp#»n u<ip(\ follftwlnq the workshot) and with each of the nine 
purchasers who had used the units and who had indicated on the question- 
naire that they would be willing to discuss the units by telephone. 
The tenth purchaser who had used the units wrote on the questionnaire 
that she did not feel that she had used them enough to give intelligent 
input. 

Each user was asked a standard set of six questions during the 
telephone interview. The essence of the user's response to each question 
was transcribed by the interviewer. A summary of responses to the tele- 
phone interview questions is presented in Table 4 on Page 24. A more 
detailed listing of the responses by each user is contained in Appendix 
C beginning on Page 61 . 

It can be seen from Table 4 that, of the 16 users interviewed 
by. telephone, 11 had used the Set ting Goals unit, 10 h^d used Anal yzing 
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Table 4 

su^H^RY or hesponscs lo tclephone interview questions 



1. Which uniU did you uv.i;'? 

<h"Mii lO^AnaVjtdnj VjK>hlctns} 

In what context did you use them? 
Vi fi'hool d'cti'ic't uK'ivkchop, 

3. With how i.iiiny and wf»at type of people did you use thom? 



9 DeHving ObjeoHvea 



4 Graduate course in education 



(Aii>}\)x) i'^'hi^-jl po'scnr.c'L: adininicti'atoro, 
'aa\!iruni ty 'i'CJ enta Hves 



100 Gmduate otudmta 



4. In what woy did you use the units? 

rmdifU\itionci 



H SpJ,p.(itnd moLteviala 
and aaUvitics 



3 No hlatdvials: 
coticepto end 
ideas only 



5. How would you evaluate the units based on the participants' reactions on 
yours? 



i'i Favomh'h 



1 M-ixed 



2 Use woe too 
non-o tructured 
to tell 



1 T oo early 
to tell 



6. Are the participants using the materials or training in any follow-up 
activi ties? 



6 Jluvc' i)i*ittcK or 
"wilL u'>fLtc diotrict 
afid/ci' oouri'.c joalo 
and (joui indloatova 



2 Identifying and 1 Yea (but ape'jifia 
pi^ioritizing activities not 

problem described) 



7 No 



*For all itcnr, (?xcfpt Itci) 3, the ^lumbers beside each response category show 
the nunib(.T of intervict^cs whose response to the item fqll into that category. 
Tho six questions were opcn-endod, and the response categories were constructed 
by the evcilujtor after the interviews to summarize tho interviewees' responses. 
Iho tot,il nuii'bcT of responses varies by item because individual responses to an 
iltnn could fall into more than one category if tho interviewee used more than 
one uriit. 
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jprpbioms, anJ nine had used perl vjjig Ob j ec t1 ve s , Twelve of the 16 users 
had used the units with school -district personnel, and the other four 
had used them in graduate courses in education. One or more units had 
bten used with approximately 54? school personnel, including community 
representatives and high school students, and v^ith 100 graduate students. 
In most instances, selected materials and activities or an otherwise 
modified form of a unit was used instead of the entire fot of unit materials 
and activities. Twelve of the 16 users (75%) reported that reactions to 
the unit(s) were favorable, and none of the users reported that reactions 
were primarily negative. Follow-up activities-mainly the writing of 
goals and goal indicators— have been or are being conducted at nine of 
the 16 sites where the units were used. It is not indicated in the table, 
but the evaluator inferred from information obtained in the telephone 
Ir.tcrvic.;; and from FWL's list of purchasers that multiple copies of 
the units were purchased by only one of the users interviewed by telephone 
and that the other interviewees who used the units reproduced their 
materials from the single copy of the units which they purchased. 
Interviews with no n-users Telephone interviews were held with a total 
of five workshop participants and purchasers who had reported on the 
questionnaires that they had not used the units and did not intend to 
use them. Each purchaser among the non-users who were contacted had 
indicated in response to Item 1 on the purchaser questionnaire that he 
had reviewed one or more of the units. The purpose of the telephone inter- 
vic\/s with non-users was to attempt to determine the major reason(s) why 
individuals who were familiar with the units did not plan to use them. 
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The five non-users were asked a set of four questions each in the 
telcphoMi.' interview. The questions are listed below. 

1. Did you order the units (or attend the workshop) with a 
particular use in mind? 

2. If so, what wds it? 

3. If you had a particular usr in mind, were the units appro- 
priate for it? 

4. Why didn't you use them? 

The essence of the response to each item was transcribed by the inter- 
viewer during the telephone interview. 

Of the five non-users who were interviewed, three reported that they 
had ordered the units in order to include them in collections of resource 
materials for planned or ongoing professional activities for which the 
intervicwi'G was responsible. A fourth non-user had ordered the units to 
review thLin for possible use in a planned summer workshop. He considered 
the units lu be appropriate for use In the planned workshop, but he 
decided not to hold the workshop because of the press of other matters. 
The fifth nun-user had ordered the units for review purposes with the 
possibility of using them for in-service training conducted by a state- 
supported regional education service center. He reported that his agency 
would use the units in the future if an appropriate request for training 
in the areas covered by the units is received from the school district. 

Based on the limited number of telephone interviews with non-users. 
It appears that the primary reason that individuals who were familiar 
with the units did not use them for training purposes was that these 
individuals had not intended to use the units as training materials. 
Halher, U. :y hod ordcM Cvi th.: units to add than to professional collections 
of rosourco nialcriels. None of the nor.-users intej'viewcd by telephone 
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Indkuted that thoy had had an unfavorublu reaction to the units and had 
not uucd theia for that reason. 

Si te y i si ts _and Us er Reports 

To obtain more detailed infonnation on use of the units than was 
yielded by the questionnaires and telephone interviews, the ev<i1uator 
visited two sites and obtained reports on use of the units from persons 
at three other sites. These five sites included an approximate total of 
406 persons with whom the units had been used for training or for program 
planning. A brief description of use of the units at these sites Is pre- 
sented belov/. The three reports submitted on use of the units and a longer 
description for one of the two sites visited by the evaluator are contained 
in Exhibit F.. Unit-related activities at the second site visited by the 

report than the brief description presented below for this particular site. 

Personnel in the San Mateo, California, City School District have 
used the procedures from the Analyzing Problems unit to Identify and 
prioritize instructional problems in the district's five middle schools 
comprising Grades 6-8. The problems have now (December, 1974) been 
identified and prioritized, and middle-school personnel are presently 
engaged in activities designed to Identify and Implement optimal solu- 
tions. The project is coordinated by the district's Assistant Superin- 
tendent for Instruction, one of six administrators from the district who 
attenvlod the August workshop, and is being conducted by a 25- person com- 
mittee?. Rather thnn using the exact procedures or the materials from the 
APiJyf^i''3-ilCo]?L-!'is unit in the project, the project coordinator and other 
district personnel who attended the workshop developed a sliglitly modified 



version of the procedures for use In their own district. The project 
coordinator was very enthusiastic about the success of the procedures as 
implemented in his district. The external evaluator who conducted a 
site visit to San Mateo, was also favorably impressed by the quantity 
and quality of the work that had been completed In the district since 
the August workshop. 

At the second site visited by the external evaluator, a team of 
three participants in the August workshop had subsequently conducted a 
two-day workshop based on the units for all 26 school administrators in 
their district. The three workshop participants, each of whom was a 
building- level administrator, selected concepts and procedures from the 
units which they felt would be most useful In their district and used these 
conrepts ami procedures as the basic instructional content for the work- 
shuji. TiiO itfiior udiiiinlstrator among the three trainers Indicated to the 
evaluator that the trainers believed that reaction to the district work- 
shop had been favorable. Two of the three trainers are also using the 
content of the Setting Goals unit as the basis for having faculty members 
develop course goals for each new course in their schools. A report 
for this site is not Included in Exhibit E because, in the evaluator's 
opinion, unit-related activities at the site were not extensive enough 
at this time to warrant a description that is more detailed than the one 
presented above. 

The Dnporia, Kansas, State College Teacher Corps Project has involved 
u<;c of the three R.'L units by personnel from eight rural school districts 
and two large-city school districts in Kansas. Initially, a total of 
3b superintendents, principals, teachers nnd community representatives 



from six districts were trained with the three units during the 1972-73 
school year. The procedures outlined In the Coordinator's Handboolc were 
followed rather closely during this training. Participants from four of 
the six districts in the initial training subsequently used the units to 
train all of the teachers and wary coimnunity representatives in their 
districts. In two of the districts, goals and goal indicators based on 
the FWL units have been developed for the schools and local committees 
are developing curriculum objectives related to the goals. In addition 
to the six districts involved In the Initial training with the units dur- 
ing 1972-73, personnel from four other districts received training with 
the units during 1973-74 and are now using the units for training and 
program-development purposes in their individual districts. Rather than 
purchasing multiple sets of the units from FWL, the users from this 

nwjor criticism of the units by users from this project has been the amount 
of time required by activities related to the hypothetical school districts 
In the units, and project personnel have substituted activities related 
to their local schools for those involving the hypothetical districts. 
In all, approximately 205 persons have participated In the training and 
prog ram- development activities under this project. The general evaluation 
of the units by the project coordinator is favorable. 

Under the administration of the Rhode Island Teacher Center, the 
rWL units have been used as the basis for five training courses conducted 
at local school district sites in Rhode Island. A total of 126 school 
administrators and teachers attended the training courses. Three hours 
of graduate credit were awarded for successful completion of a course. 
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Typically, all three units were used in their entirety in a training 
course. Following the tralninfj courses, participants from four of the 
five districts have used the content and procedures from one or more 
units as a basis for working on issues or problems in their own district. 
The evaluation of the training was favorable or very favorable at four 
of the five sitci and unfavorable at the fifth site. The unfavorable 
evaluation was attributed by the Teacher Center representative to poor 
performance by the course instructor, rather than to the units themselves. 

The Settintj Goals unit was used by the Region VII Service Center in 
Kilgore, Texas, to train 24 school administrators and teachers in Marshall, 
Texas, Independent School District. The unit was administered during the 
1973-74 school year to assist district personnel in the development of 
priijcAitlon^l goals for the district. The training coordinator used most, 
bul iiol cill, of liie activities from the unit with the participants. Con- 
current with the training, participants applied ttie procedures from the 
unit to develop a set of prioritized goals and goal indicators for the 
Marshall School District. Personnel from the district are now engaged 
in developing performance objectives based on district goals. The Service 
Center Coordinator reported that the Setting Goals unit was very effective 
in the training course and program-development activities. 

Suninary 

Information about use of the units v/as collected through mailing of 
a brief q'jostionnaire to all participants in the August workshop and to all 
persons other than librarians on FWL's list of purchasers, holding tele- 
phono ihlcrvicws with individuals who indicated on the questionnaires 
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that Ihey hdd u&i?d one or more units with others, and conducting a site 
visit or obtaininu a written report from five sites at which the units 
hod been, used rather extensively. Twelve of the 25 workshop participants 
(48%) and 10 of the 61 purchasers (16%) who returned completed question- 
naires had used one or more units with others. The users Interviewed by 
telephone rvpresented 16 different sites and had used the units with 
approximately 5-12 school personnel and with 100 graduate students. Unit- 
related follow-up activities involving the writing of goals and goal 
Indicators or the identification and prioritizing of problems were being 
conducted at nine of the 16 sites represented In the telephone Interviews. 
Rciictions to the units were favorable from 12 of the 16 sites, and none 
of the 10 users reported reactions that were primarily negative. Inter- 
views with a small number of purchasers and workshop participants who 
h?d not I'^ofl thp units with others Incilcatefl tnat mosi: oT mese «on-uber> 
had ordered the units as resource materials for professional collections 
rather than for use as training materials. 
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4. EXTCRNAL REVIEWS 

Ihree professional oducatbrs with expertise In areas related to 
the training units were identified by ITC and approved by FWl as potential 
reviewers of the units. Arrangements for reviews of the units v/ere sub- 
sequently made by ITC with each of these three individuals. The three 
reviewers wore Dr, Keith Acheson, Assistant Chairman of the Department 
of Curriculum and Instruction at the University of Oregon; Dv. Norman 
Boydn» Dean of the School of Education at the University of Cdl1forn1a» 
Santa Barbara; and Dr. James Zaharis* Assistant Superintendent of the 
Mesa, Arizona » Public Schools. Dr. Acheson and Dean Boyan were selected 
as reviewers because each of them has done considerable substantive work 
in the areas covered by the training units. Dr. Zaharis was selected 
because his exoerience as one of the coordinators of the Auaust workshon 
v/ould provide him with an excellent basis for reviewing and evaluating 
the units and because, as an Assistant Superintendent of Schools* he 
represented a large class of potential users of the units. Dr. Zaharis 
wrote an initial review of the units prior to the workshop. After the 
workshop he wrote an additional set of comments to summarize his judgments 
about the units based on his experience in the workshop. 

Each reviewer was instructed to thoroughly review the three units 
and the Coordinator's Handbook and to submit a report which addressed 
explicitly at least the following four topics: 

1) need for the units 

2) quality of the instruction 

3) overall effectiveness of the units 

4) likelihood of successful installation and use 

A list of 15 evaluation "considerations" organized under the four topics 
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above was aUo sent to each reviewer for optional use In addressing 

tho four topics. 

A sunmary of the three reviews of the units Is presented below 
under the tuiadlncis of the four topics which the reviewers were Instructed 
to address explicitly In their reports. The complete reports submitted 
by th..- reviewers are contained In Exhibit F. Also Included In Exhibit 
F are a description of the directions to the reviewers and a list of the 
15 evaluation considerations sent to the reviewers. 

Need for the Units 

The reviev;crs' comments on the need for the units' related primarily 
to the existence and relative effectiveness of competing programs and to 
the inportancs of the skill? taught In the FWl units. With regard to 
cor|u;liny piogfW.s, the reviiwers varied considerably In the number of 
programs they mentioned and In their comments on the effectiveness of 
the programs relative to the FWL units. One reviewer cited 15. separate 
publications In his discussion of competing programs, but many of these 
publications do not appear to be closely related In purpose or content 
to the FWL units. Competing programs which were mentioned (by title or 
developing agency) by at least two of the three reviewers are the SPECS 
program (School Pl anning^ Evaluation and Communication System ) developed 
by the Center for Advanced Study In Educational Administration at the 
University of Oregon, the Co mprehensiv e Planning In structional System 
dovolnppd by Kcsr.^rch for {?cttcr Schools (RBS), and the Res wrchJJtlJJiljia 
P,.(,sir.„_s.-,lvin(j pro-jram of the Northwest Regional Educational laboratory. 
Overall, the rpvic-wors' coimicnts indicate that there are a number of 
priUji.M.. tliat ccin be considered to be competitors to the R^IL package. 
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Am(«fuj the revicv/s thuro was only one rather thorouyh contrast of 
the V)hI units with another program— a fact that may be due to the 
difficulty of nuiking direct comparisons of the overall valuy of programs 
which can he considered to be coii.petitors but which nevertheless differ 
apprpciiibly in purpose, objectives and content. This contrast Involved 
the FWL units and the SPfCS program. The reviev^or concluded that the 
rWL packiKjc Is preferable if the users wish to concentrate on Instructional 
plannlnij, but that SPECS is preferable if the users* major concerns include 
a "broader range of educational planning behaviors across an array of 
administrative and instructional positions." This reviewer also stated 
that there is mOre substance in the reading sections of the SPECS program 
than in the FWl materials. Another reviewer rcn^arkcd that the FWL units 
do not hvi\o as much content depth as the competing RBS and American 
Ir.rtit'.:to for Rorcrch m^lfTl?ls ?»nd that they flre not as comprehen? 1 ve 
as mult:rials developed by Roger Kaufman and by the Northwest Regional 
Laboratory. However, this reviewer also stated that the Analyzing Problems 
unit describes a process that is badly needod by the public schools and 
that he knew of no other set of materials that attempts to set up such 
a thorough process of validating problenis. No other direct comparisons 
of the FWL units with competing programs were made by the reviewers. 

All three reviewers stated without reservation that the skills taught 
in the FWL package are very important for school administrators and/or 
professional educators involved in instructional program planning. 

Rcvuv.'cr corm-nts on the quality of instruction related primarily 
to the iippropriflt'^ncss of the instructional activities, the appeal of the 



i>idtt?rii»K und acti vitlcr., ciiul the iippropriiiUjiUii.s of the asscbsiiient 
cullvUifs. With rcijard to appropriateness of the instructional activities, 
tvo of tho three* revlev/urs made cominents about the threo units collectively 
which indic<!tcci that tho reviewers felt the activities and exorcises arc 
cjt nrrtilly i ppropriato. These same revie-wers also remarked that tl»e acti- 
vities urt ipod for pronuting interaction m\ou^ user^>. One of the:.e tv/o 
reviewers reported thot the exercises and simulations are realistic and 
that Llie iM livities raised problems and Issues that are widespread in their 
applicability. The third rcv1©r/cr commented on the instruction on a unit- 
by-unit basis. He was most positive toward the P rob lem S olving , unit and 
I past positive toward P?j:1jij}a ObJc^cXives TV\'o of the three reviewers 
wrote a m-siibcr of specific criticisms of the units on a unit-by-unit or 
a rCule-l'.v -ti^Julc bc<;i?. Overall, thp roni.npnts of one of the three 
revii.v.'sM :. rt'jdf dinfj i!ia cipp'opriuUfiesi of Initi lictloii acrosc the thvce 
units were quite positive, while the comments of tho other two reviewers 
indicated only a fai:- or rather neutral appraisal of the instruction. 

The comments on the appeal of the units were mixed. One reviewer 
thought that appeal of the instructional activities In Units 1 and 2 is 
on the high side, but he was disturbed by what he felt to be unnecessary 
efforts to be cute, joking and overfamlliar In Unit 3. A second reviewer 
found the sequence and type of activities to very appealing, but he thought 
that the materials are too bulky. His concern over bulk was shared by the 
third reviewer, who stated that the materials are wordy and not appealing 
C(.)>-. jtiv-.;! I Two or the ti'.roe roviewerj sui:'>:stcd pcickcujing the rcciding 
portion or '0\r units ut-pv-irr-tcly f\ou\ the ether uMcri Is. 
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The rcvii'v/ers ' coiiminMis on the upproprivilunuSi. of the assossntont 
were rcfemicud prlnsarily to the self tc^ts In the units. One rcvitwor 
reported that that tostr. sccw to bo adequate and that they provide sood 
practice for the user. Another reviewer indicated that the tests ere 
adeviuato for a user to ijain the feeling that he has mastered the walorial 
and that Ihcy aUo serve to stimulate discussion and provide varied learn- 
ing activities. The third reviewer felt that a clear relationship is not 
t'SUblished bttweon the stated objectives of the modules and the self- 
test itetii^. He was also quite concerned about the lack of depth in the 
su»j{jested responses to the self tests. Only one reviewer commented on 
tho issue ot wtiether an individual or team that has used the materials 
could subsequently set appropriate goals, handle a real problem or derive 
obj'jctives. The reviewer noted that an answer to this Issue can only be 
cornocttjrcci . 

Overall Effectiveness of the Units 

In considering the overall effectiveness of the units, the revi avers 
generally directed their remarks to the amount that users are likely to 
learn, the major strengths and weaknesses of the units, and an overall 
evaluation of the units. The three reviewers differed considerably from 
one another in their judgments of the overall effectiveness of the units. 

One reviewer was very positive in his assessment of the overall 
effectiveness of the units. He reported that the exercises and activities 
app'Air to rncurp that consciontioir. participant'? will know how to perform 
tiu' funrtif.iS ."r.d proccjscs covered in tU? unitr.. He felt th?t the 
slrcfi'jthr. of the materials are in the power of the Ideas and processes 



and in thfir foriiKtt, Irarnlmj activities and attructiveness. As weak* 
nesscs, ho citui the existence of many competing products which cover 
similar or related ideas, and he noted that the bulk of the materials 
iriay deter so:ne projiptctive users. He thought that the materials are 
woll-plcinned and in tune with current thinking and enlightened practice. 

A socond rcvicv/er was very concerned with the nature of what users 
will lec»rn from the materials. He felt that the materials are much 
strojjgtT in providing practice at working with colleagues as a team to 
determine instructional purposes than they are at providing a sound knov/- 
ledge base for users. In this reviewer's judgment, the reading portions 
of the materials provide relatively little substantive knowledge that is 
useful for dealing with the difficult issues involved in the process of 
determining instructional purposes. He thought that the units may increase 
tt\o nrnnrienry ot n<;f»rs in rippiymg ronvpnriona i wisdom and pxperienrp 
to the coinplex issues in determining instructional purposcs—a result 
which, he noted, is desirable— but that they are much less likely to raise 
the level of substantive sophistication of users in determining purposes. 
In his judg.nent, the units are much stronger methodologically than con- 
ceptually. He also felt that the Coordinator's Handbook does not contain 
sufficient substantive content for the coordinator who lacks a strong 
substantive background. 

Despite the second reviewer's concerns regarding conceptual content 
of the units, he reported that his overall evaluation of them is favorable. 
H? sr.'i tho units as being valuable, not for use in an isolated or free- 
f.tnnrijor! ri.ir.:K'r, but as the brv^is for a practicum type of experience 
aciorpcf.yinq a course which covers in depth the substantive content related 



to ihe procesu of tloterwiinlmj instructional purpcsc^i. He feels that 
tht^ jtd'Jitional suhctantivo con lent of such a course Is hcucIchI to provide 
educdturii with in:;-urta(it knowlfdtje not cental n(?d in the units end that 
the units v/ould be va1us\ble in providing practice in the actual process 
of uLtciinininy ins true tloni»l purposes. 

The tfii rd reviewer thought IhcvL the ,%tjjng^ Goa^ls and /|y^^ 
Jl>f*yf''l'l i'"it.s are strong with respect to their content and are likely 
to bp effective v/ith u:,ers who are highly motivated. His evaluation of 
the ovcrcill effectiveness of the D eri vim: 'b jecti yes unit Is less posi- 
tive than his evaluation of the other two units. He cited as major 
vvfakn(L'Sscs the wordificss of the materials and the lacl; of variety of 
forii.'t in bulh the roterials and the instructional activities. Because 
of tliese pcrccivcJ ..•Cr^knossor. , he did not feel that the materials will 

(..Hpc+iv^- •Trr*; v.'ho )?.rk ? strony • fieri re to acquire the kncv/led'jc 
and skills which they cover. He reported, however, that he docs reco;imend 
use of the units by his own personnel. 

Likeli hoo d of Successful Installation and Use 

The final general area which the reviewers were Instructed to address 
was the likelihood of successful Installation and use of the units. Re- 
marks in this area were directed at the likelihood that administrators 
trained with the units will use the materials and skills In their work, 
the potential benefits of the units if they are used In school districts, 
the likt lihon:.' thM. p. porr.'.in truin-.i with the uatorials can train olheri 
si.rt: '•.dly, V.c cost of tlio units, and t.h? amoi'iit of both inc. true lur 
and usfv Lime rcquiiod to the units for traiin'ng purposes. The 



conruMiU of the three reviewers on the probi*b1e [ifa of tho utiits dealt 
with tho context and manner in which the units ore likely to be usf:d, 
rather than with the more speculative issue of whtther they aJ^'C likely 
to bv widely used. None of the reviewers coiiwentcd directly on this 
latter isue, which is a very important matter in making decisions regard- 
ih';) future disr.riii nation of the units. 

( :ie nvi(ver thought that the units will most likely be used in 
in service pro'jrat:is, extension courses, and suimier courses for sc[iool 
admiirlGtrators end teachers. He reported that they are also appropriate 
for use with faculty committees preparing to work on instructional plan- 
ning proble;:is and in graduate- level university courses' in school admini- 
stration and in curriculum development. In this reviewer's judgment, 
u*ie of the units in school district will undoubtedly result in desirable 
chTij '^ in the Htstrict's 1nstn«ctiona I progrntn. He believp'J that p 
bi!;.kgruund of relevant practical knowledge and experiences, as well as 
teaching skill, is very important in order for a person trained with 
the units to use them successfully to train others. This reviewer cited 
$5.00 per unit as a desirable maximum cost for use of the units, which 
presently sell for $8.95 each, in in-service training and in university 
courses. He felt that the preparation time for an instructor to use the 
units will be minimal, once the instructor is familiar with them, end 
that the total training package Ct.n be completed entirely or in large 
part in the amount of class time normally available for a university 
cnjrs;^ or for an extension course offered in a school district. 

A second reviewer believed that tf.e skills required froM the R-M. 
unit, are likely to be used in a selective manner by adiiiinistrtitors who 

-39- 

erJc •\?> m 



are trained with the uriU^» but: wi in tiij same order or to .the? extent 
etivl*;iunod ly Uk» unit dovplopors. He IndUdtod thai, In his judgiuent', 
persons previously trained with the units would be li!;oly to refer to 
them frequently tor tjuidance tind nsslstdncc. This reviewer felt that 
u*ie of thf! uaitb by cidiiiinistrraurs will result in improved skills in 
Korkinrj with oUior;>, sonsc Increase in subs tan Live knowledge, and o 
conc.wdUnt increas-c in capobilitics for tinalyzino issues «nd problems 
in if.structional planniny. like the reviewer cited immediately above, 
he believed that the prior experience and training of a person will have 
a very important effect on his success in using the units to train others. 
He did not think that training with the units will adequately equip a 
person who is not beyond the level of the materials to train others 
succi'Si. fully, lu this reviewer's opinion, cc'.t vMll not be a determining 
factor in i-ro of tlie unitr . He reported th-'-t thr; ?»iount of in^^tructor 
time required to use the units is not excessive and that the amount of time 
required of users is reasonable. 

The third reviewer felt that the most appropriate uses of the units 
are in task force training, training programs for potential administrators, 
graduate-level university courses in educational administration, and 
general administrative in-service programs. He was not optimistic about 
reactions to the units in the latter type of program, hov/ever. In his 
judgment, the Setti ng Goal s and Anal yzing Probl ems units are more likely 
to be used by reviewers than the Deriv ing O bj ectiv es unit. He believed 
th::t I >ri; n'idls are generally available on Instructional objectives 
t!;.<n on V>p. ether top1c<i and that the Ponvi!Vi_pjKiec li^^^^^^^^ doiT-. net 
c(wi|»iir»» favoi'dbly to bom; of the evr.ilable naterials . 



P" Rovlws of tho ni!.t,f;.rnili.iiiig. Jjj.$^^^^^ r'.urp.p.f;<;5 units were obtained 

ill from tlirec proft'ssionul e<iuctitors, two of whom have clone considerable 

sub«;tci:Uve work in tlio arcai covcrod by the unUs and a third who had 
served as a coortiinotor at the August workshop and who represented a large 
clots or iJOlentiaT u^^crs of the units. There ivas considerr.ble variation 
in the nature of the comments of tlie three reviewers and ' i their reoctioiis 
to thf units. The ro«iclion3 of one reviewer were consistently very favor- 
^ able, except tliet he expressed some concern over the number of competing 

L procjrfUiis and the bulk of the units. The comments of the other fevo 

^ ruviwers were less favorable, even though each of these reviewer: reported 

that ho would rc?cor;ini!i»nd use of tfie units for particular training pur- 
j^jit •.. ar>socio.led v/Uh his own work. One of the latter two r^eviewtrs 
goRcrally reacted' favorably to i-ho SeUinO-.QojJ.s tin-' MPL^^^^ 
units and unfavorably toward Deriving ObJecti ^. Overdll, It seerns 
most accurate to describe the reactions of these two reviewers toward 
the three units collectively as being neutral to slightly positive. 
All three reviewers reported that it is very important for school admln- 
instrators to possess the types of skills taught In the units. 
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b. CONCLUSIONS 



f" Two purposes for an externa] evaluation of the Determining Instruc- 

tional Pur()oscs training package wore cited in tho Request for Proposal 
issued by tho Far Weut laboratory. The primary purpose v/as to provide 
infonuation and recwnnendations for use in making decisions regarding 
dissenjinrttlon of the units. A secondary purpose was to provide infor- 
mation that is useful to school administrators in making decisions 
about purchase and use of the units. 

The external evaluation of the units was conducted by Instructional 

_ Technology Consultants under a contract with the Far West Laboratory. 

Infonnation related to the two purposes cited in the RFP was collected 
through use of several different procedures. An intensive four-day 
workshop, ill which the units were used to train 25 school administrators 
and university faculty members, was held to obtain data on participant 
perfoiwiice and alliludes following training with the units. Informa- 
tion on use of the units was collected through questionnaire surveys of 
the workshop participants and of individuals who had purchased the units 
from FWL, telephone interviews with persons who had used the units to 
train others, and a site visit or obtaining of a report from a key user 
at selected sites where the units had been used rather extensively. 
Additional evaluation infonnation was collected in the form of written 
reviews of the units by three professional educators, tv/o of whom had 
strong substantive qualifications in the areas covered by the units and 
a third who had served as a coordi^^atcr at the four-day workshop and who 
represeiittsl a largo class of potential users. 
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On the basis of the information collected ifi the external evaluation, 
It appears to the cvaluator that five* general izntions can be wade that 
should be useful in makirivj decisiuns about dissemination or about pur- 
chase and use of the units. Each of the five generalizations is pre- 
sented below with an accompanying description of the data and evaluator 
judgments which support the generalization. 

1 • te.ct 1 pjjsjpf , fu ture users toward the u nits are li kel y to be favorable . 
The data collected during the external evaluation indicated that 

« 

attitudes toward the units are generally favorable. The attitudes of 
the workshop participants toward the units were consistently favorable, 
even though presentation of the units In Intact form in an intensive 
four-day workshop undoubtedly was not a highly desirable method of using 
them with trainees. Persons who had used the units with others for 
Ifoiniiivi ur pi uiji aiii-uevcloi^iiuiil puipu^ea dl^u I t:pur leu coiid ibleiilly 
favorable reactions to the units. The collective reactions of the three 
reviewers were less favorable overall than those of the workshop parti- 
cipants or persons who had used the units with others. However, one of 
the reviewers was very positive in his evaluation of the units, and each 
of the other two asserted that he would recommend their use for particular 
training purposes associated with his own work. 

2 . In most instances the units are not likely to be used as Intended 
by"the devolb pbrs ." " 

This statement applies both to the purpose for which the units are 
used and to the manner in which users proceed through the materials and 
activities. From the interviews with users of the units, it was evident 
that many school personnel want to use them directly for the purpose of 
applying the processes covered in the units to issues or conditions in 



thoir own school districts, rather than for the purpose of being trained 
In how to perfonn the processes so that they subsequently will be able 
to apply them in the schools. They want, for example, to use the Setting, 
Goals unit to set goals for their districts, not to learn the process of 
setting goals per se. This was a manner In which several of the users 
who were interviewed had used the materials, and In the evaluator's 
opinion, It is an appropriate use. Because the materials can be readily 
used as a guide during the process of setting goals (or during the other 
processes), it seems unlikely to the evaluator that previous training 
with the unit and practice of the process for a hypothetical school 
district will result in a considerable improvement in the goals that a 
planning team sets for its own district. 

The Rianner in which users proceeded through a unit also typically 

^Uft I Wm I I will wilU pi MUwwi • w wv% iii wAiW WwiUill«%i#vl ^ iftv^iiMv • S^%%%j 

three of the 16 users interviewed by telephone reported that they had 
used one or more intact units and had closely followed the procedures 
from the Handbook. The most common practice was to use selected materi- 
als and activities from a unit rather than the entire unit. Another 
frequent modification was to substitute real Issues or problems from a 
local school district for the content on the hypothetical school dis- 
tricts in the units. 

3 . The imre .gop^riX JPJ'."L'' !?>lL outcome for each unit is the m ost Impor- 
'i ns true t ior.V: 1" bir jc'ct i viVTor" th"c"uni t . 
One or two rather general outcomes, or goals, is implicit in the 
instructional objectives and activities for each unit. As noted in Chap- 
ter 2, for example, the general outcome or goal for the Settin g Goals 
unit might be stated as follows: 
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Derive one or more yoal sldtementb for a school district 
and a set of prioritized indicators for each statement. 

The general outcome for each unit involves development of a product 
that is the culmination of the various unit activities. Development 
of the product typically requires use of rules, definitions and criteria 
included in the unit and serving as the content for many of the Instruc- 
tional objectives of the unit. Many objectives simply require recall 
of content In the reading section of the module. Yet there is no need 
for the trainee or other user to commit this content to memory because 
he should be able to refer directly to it In the materials whenever neces- 
sary and because most of it is not generally useful In situations other 
than when he is working with the processes described In the units. Many 
other objectives involve the application of rules or criteria to make 
decisions that are very often a matter of personal interpretation (e.g., 
"Does the set of goal indicators include indicators for all Important 
areas of behavior related to the goal?"). It is Important to list this 
criterion for users to refer to and to define "all important areas of 
behavior," but there is little value in considerable explication or In 
practice related to this particular criterion because the explication 
and practice do not result in attainment of a skill that can consistently 
be applied to t^^ake correct decisions about useful examples. The "correct" 
answer is a matter of interpretation for all examples that are not written 
so that the answer is obvious. 

In the evaluator's judgment, the general outcome for each unit, as 
contra-.tcd with many of the instructional objectives, represents an out- 
come th;'t is very i»:iportant tu many school personnel. The procedures 
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described in tlie units, particularly in Units 1 and 2, are appropriate 
melhod'i for useis to follow in order to attain the general outcomes. 
Users workiny toward this <joal should be able to refer directly to the 
procedures in the units. Direct reference to the materials obviates 
the need for users to attain the instructional objectives that are based 
on rectTil. and there is little value in practicing examples related to 
objectives that require decisions based largely on personal judgment. 
Thus, it seems appropriate for users who want to do so to use only the 
content of a unit that directly involves procedures for attaining the 
general outcome of the unit and to by-pass the other reading material 
and the self-tests. Participation in the simulation activities would 
depend on whether the. user was applying the process from the unit to his 
own school district or to the one of the hypothetical districts from 
the imt«;. 

The above generalization regarding the relative importance of the 
general terminal outcome of a unit, as contrasted with the instructional 
objectives, appears to conflict with statements by all three reviewers 
that it is very important for school administrators to possess the types 
of skills taught in the units-at least to the extent that the reviewers 
were referring to the skills represented by the instructional objectives. 
On the other hand* the generalization and the discussion of it are con- 
sonant with the strong belief expressed by one reviewer tliat the method- 
ological content of the units is sounder than the conceptual content in 
the reading sections. 

4. The r»:v''l'"- ^"IT•i)J•..?M co?.t.P/-J-!)P-l*-0Jj^?-ir'?.-"9A^-^^^^ 

The present cost of the units, approximately $9.00 per person for 
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each 10 la hours of instruction, i., hlyh ri.lative to the cost of most 

instructional ridterials. In cases where individual users must pay for 

their own materials, the relatively high cost inay often affect use or 

sales of the units. This point is illustrated by the fact tliat multiple 

copies of the units were purchased for use at only one of the 16 sites 

represented by users intervicv/ed in this study. The other individuals 

who used fiiuUiple copies of one or more units purchased a single copy 

and reproduced copies from it. Of course, it is particularly easy to 

reproduce the materials in their present format because they are loose 

leaf and 8'^." by U" in size. Furthermore, users are less likely to 

be willing to pay the full price for a unit when they use only selected 

materials from it, as they typically did in this study. 

S« U .-t-Mj^Pkis i.on..i s livid e^ to_^ubli>h o\ul di sscnn na tejhe. uni^i.»-J]ii 
laBora to ry s nbul d coiis i (ior" tfTe " d'c'?i i raFHT L;^ oT^jJcj^^^ Uie" De r" fv i nc| 

Two of the three reviewers expressed rather strong concerns about 
the Deri ving Objectives unit. One of these reviewers reacted quite 
favorably toward the first two units but consistently unfavorably toward 
Deri v i ng O h j oc tiyes , while the other reviewer was bothered considerably 
by what he judged to be ill-conceived attempts at humor in the unit. 
One reviewer also noted that more and better competing products are 
available on the topic of developing Instructional objectives than in 
the areas covered by the other two units. The evaluator shares the per- 
ception that the Deriving Objectives, unit is not as' strong as the other 
two uni t-i . 

The part of the Derivin':i Objc'ctives unit that did consistently re- 
ceive fnvorablc conincnts from the reviewers was the programmed text. 



If the dochion v. tnule to i.ot publlsti and dlsjciinnute the unit, con- 
sidff^nion rould be (livun to iiiurkoting the procjranmod tuxt as a separate 

The imi.iiy. C(V!ls and A.i<Llj?jniJ.PXP!?.V^'}5 units tocjc?thcr have enough 
content that they ccuUI sorve as the basis for a three- credit-hour 
university course. It also seems that most school administrators are 
n»:»ro likely to wcrl: at tho tcn'.ks and the level of generality from these 
units than at tho so»hat more detailed level required to derive objec- 
tives. The present coiiiblned cost of t\70 units, while still on the high 
side for u university course, would not be out of line with bool'x costs 
for many courses. Printing the reading content and self- tests separately 
from the team activities in each unit, as suggested by two reviewers, 
could provide more flexibility with respect to cost and use of the materi- 

activitic"i to have the potential of being used as a free-standing com- 
ponont. 

The process of making a decision regarding the publishing and dis- 
sciin'nation of instructional products involves a number of uncertainties. 
There Is no set of well -developed rules for accurately judging the 
desirability oF disscfiinating a product when a clearcut answer is not 
available to the question of whether the product should be disseminated. 
Hopefully, the information contained in this report will be of use to 
the Laboratory in arriving at a decision about publishing and dissemination 
of the Detor-p ini LP.?A.*'y5'!^'^.''*^L]ll'*inPJ^>l^^^ units. 
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TrST-DtVElOPMtia PROCEDURES AND SAMPLi: UNIT TtST 



Iwo test foriii-i {Fonii A and form B) wore constructed as unit tests 
for well unit. Tho initial sttp in development of ecKh unit t(?3t was 
to develop n pool of nailtiple-cholce test items -keyed to the instruc- 
tional ohii-ctivcs cf the unit as stated in the materials. Less than 
one Iti'.ii per object ive was constructed for each unit prir.arily bec«=iuse 
of (1) concern over the amount of testiny time that v/ould be required 
durinq iw, workshop and (2) the fncl that sonic stated objectives could 
be lubsuiiied for assessment purposes under other more comprehensive 
ohjeotives. The mir.-.bcr of oUjcctives and test items for each unit are 
shown i»(.'low. 

Unit pbJiLcUves Mt Items 

1. Setting Cools 17 12 

2. Analyzincj Problems 32 22 

3. Deriving Objectives 13 12 

Each test item was written to assess performance on one or more 
instruct io?iol objectiver. stated in the materials. The Instructional 
content in a enil, as presented in the reading section of a module or 
ccCvT-.ior.any in the suggested ^responses to self-test itwos, waS used 
,•»«; tiip <.niin:n Tor acneratiny iioiii contjcl dn:>wcr onJ ^»iwm»»Iu1c 
dlUracturr. lor each item. Thus the "authority" for the correct answer 
to an item and for tiio dit.tractors was always the instructional materi- 
als. If the itciii writer questioned the accuracy of the instructional 
content related to a.i item or thought for some other reason that the 
correct answer was potentially controversial, he noted the page number 
of the relevant content In the materials and wrote a rationale for 
the correct answer based on the instructional content in the materials. 

The test items for each unit were assigned by the test developer 
to Forms A and B of the unit test according to a set of criteria (sub- 
mitted to FWL in a separate document) designed to ensure equivalence of 
the t'.vo forms on a number of important test characteristics. The b/o 
test forms for Units 1 and 3 contained six items each. Forms A and B 
of Unit 2, which was longer than either of the other two units dnd 
which had many more objectives, consisted of 11 items each. 

All items for the unit tests were initially written and assigned 
to test forms by Educational Evaluation Associates of Los Angeles under 
an ayreiMicnt with ITC. The complete set of itrms was then reviewed 
by a ntatr m-vUtM- from FWL and a staff member from ITC. Many of the 
iUii.; vvcrc- revised Ly ITC on the basis of the FWL and ITC reviews. 

FollP'-'ing r.:vision of the toit items, the tv.o fo»vi3 of Ci'ch cf 
thr thi-oc unil tr-.ts were a-Js-iinis Icrea to five gradtjalo stuci^nlv at 



Arizona Stati- Univosily. llic pii rpos (?s of this tJ(hinniblratiofi W(?rt' lo 
Identity poti'iitial probU:iis with the iltiu-.> <intl to obutin estimatos of 
the rcUilivo cliffit;uUy of the tv/o foniui for each unit and cf the 
iiinoi:.'il of time retiulrcd to iom|i]eto iwch form. Minor revision;* in tho 
worUinfj of scvorul Items v/ero niaiif; on the basir. of this odministration. 

Form. A of the tof.t for Unit 1, Soft in-) OoaJ^» is presented on 
Payer. l?.-h''\ iv. an excnr.plf of a unit to:A'. Copies of both t(?st fonrs 
for i-Jt:h ot tnt^ thrro uniU, v/ith tcich item rofercncod to the nrxfuU; 
uhd objective for whicii it was written, iwo contctinerl in Exhibit C. 



SAMPLt UNIT TEST 
Test Form ^ 
SEITING GOALS 

Name ^ 

DIRECTIONS: Circle the letter beside the correct answer for each itcnn. 

1. listftd bolow are throe statciiiLnUs in the goal refinement process: 

if]. The student can find the correct sum of a column of whole 
numbers . 

#2. A well educated citizen will be able to manage personal finan- 
cial affairs, such as loans, credit cards, and bank accounts. 

#3. Ihe student will be able to perform all basic arithmetic 
operations. 

Which is the correct sequence of these statements in the goal refinement 
process, from educutionul philosophy to instructional objective? 

a. ^;'l, ^'2, f?3 

h. ff\, 

*c. ^2, /'3, n 

d. #1. #2 

. e. S2, n 



2. Which one of the following goals and objectives refers to s tudent outcomes 
as opposed to other school concerns? 

a. The faculty should be involved in the development of curricula 
that meet student needs. 

b. The school counselor should meet with each junior at least once 
during the Spring to discuss career plans. 

c. The volunteer aides should be given a training session before 
being assigned to work with students. 

*d. Tonchcrs should develop student interest in reading by letting 
students choose their own books. 



*Tht' correct response for euch item Is marked with an asterisk. 
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Which Olio of the following stotenieni-. would bi'. the mut iippropn'ato goal 
iiKlicator tor the cjcxil: 'Hrrttticf.'i. bood Financidl Habits"? 

a. MikGri purchases wisely. 

b. Knows consumer rights and responsibilities. 

c. Does not make purchosos without comparing prices. 
*d. Open; and maintains a checking account. 

e. Purchases lunch in school cafeteria rather than in nearby snack 
bar. 

listed belov/ arc the two goal Indicators for the goal of "Responsible 
Citizenship." 

#1. Roods newspapers and magazines about the point of view and record 
of each candidate before voting for one of them. 

n. Prior to election^ listens to TV and radio programs regarding 
each political candidate. 

Only itei^ bclov.' is both a criterion for adequate coverage of a goal 
uad Is not vlylated hy the above set of indicators. Which i lem is Mi 

a. RclDt1on*?hip of goal indicators to goal. 

b. Inclusion of goal indicators for all important areas of behavior. 
*c. Level of generality of goal indicators. 

d. Overlap of goal indicators. 

Once a set of goal Indicators has been specified for a given goal, which 
oiv> of the following factors is most important as a basis for determining 
the relative priority that should be placed on each indicator? 

a. Its relevance important social issues and concerns. 

b. The time, money, and expertise needed to achieve it. 

c. The feasibility of actually measuring whether it is achieved. 

d. Ihe pr;»fc:>s1onal judgment of consultants who h;u'e expertise 
in the goal area. 

*e. The ppiPion'■^ of school and consnunlty groups. 
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Ono opinion-(jathf}rip<j technique involves designing a questionnuiro for 
submission to community reprusontativus . Which one of tho following 
is a prob'jbte ddvantcige of this technique? 

*a. Appr-opriatf sampling procedures can be used with this method, 

b. The number responding out of the total number sampled will 
probably be high. 

c. It is relatively inexpensive in terms of staff and consultant 
time required. 

d. It provider, intensive feedback from each participant. 



UIKICTIOS'-.; rittisc (!o not vtIIc ycu;r niinio on th-fs form. 

Ci)iui)kU' ltf.;i.:i ]- 20 ly circlir.y a nvwhvr for eech Uen-i to indlcdto the oxtcnt 
to which you t<(j;v.o v.ith tlie st(itcr.orit. Use the lollowing key: 

1 - stronv« tsnrccrcnt with r.taterx'nt ^ ^ disngrco v/ith statcnont 

iKjiccroru. \.-itii sutcr^nt 5 « F-troncjIy dibctgrcc? with statf.n-env 

3 ncillu:r r.cjivc nor dir.cjjree 



i. 


I knt.v; of oticr niitonais v.'hich cover similar conttnt 


1 


2 


3 


4 






to thi. unit 1 jui. t ccr plctccl fncl \.'hich pre rryre 


* tl) 1 






f i'/ 






effective for trtiining t;C";iin1i,trut.orb. 




The vn\i is pctonticllv useful, for trairiinn school 


1 


2 

mm 


3 


4 


5 




ac(i'irri:Urt! tors. 


(-6) 


(C) 


(0) 






3. 


it v.-t'S difficult to dctorr.niio vmat I v/as supposed 


1 


2 


3 


n 


5 




to Ic.'rn fro;;, the i.nit. 


(0) 


(1) 




re?; 


(0) 




V\c j.l.illi-. ici.-'ht 5n ll'.o unit dre irfjortiinl for 


1 


2 


3 




r. 

V 




school o**...ir.if.trt;tc» 5 to locun. 


U) 


(7) 


(0) 


(0) 






liu: ict •,c'\;l <-:v:ti vi ticT. in thc' unit proviccu 


1 


2 


3 




5 




USr>f«H •\\-;',r.y: ^1••^rJ ? nn pr'fCtiCO CH tlin 5l'lll*J br 1 na 

teuf i t. . 


(V 


(P) 


• ^ • 




fn) 


6. 


Iho ini.tructici.rl iiCtivitits were at about the r1c|hl 


1 


2 


3 




5 




level of difficulty. 




(6) 


(S) 




(0) 


7. 


The sano skills tculd be IcarnGd in far less tiir.o 


1 


2 


3 




5 




than tiii; unit presently trkcs. 


(V 


(2) 




(5) 


f(7; 


8. 


In gcpfira! , I liked the instructional activities. 
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2 


3 


4 


. 5 






(0) 


(8) 


(2) 


a; 


(0) 


9. 


A diff- rent forn of practice end asf,c:siier»t would 


1 


2 


3 


4 


5 




have Irjn; prc.-crtblc to the self-tests in the unit. 


(0) 


(PJ 


0; 






10. 


1 lerrncd a lot frc'". the unit. 


1 


2 


3 


4 


5 






(0) 


(?) 




(2) 




n. 


1 CO r:ct plan to us?* either the n^Qterials or the 


1 


2 


3 


4 


5 




tri'iinr;ti 1 received in ry school or district. 


(0) 


(1) 


(2) 


f5; 


(p) 




Coi"r-rc rf \;iti otl.t'r or;,;ir;i'-itrriti vo training I have 


1 


2 




4 


5 




hiVi, 1 \.:uld »: .c- thli uriit very le\/. 


(0) 


(0) 




0) 




• %* • 


1 v.i-!:I-.; r<'visi. ( O.-iv v.-iio hflvo > «- :-.r.or.:-iM 1 i ; y fcr 


1 


7. 


3 


4 


5 




ir. .Ir.c' if . . ] p ^t'^r.-.;;. plcnr.ing lo attirKi a •./.;*!;$.'. Ci-' 


(V 




(2) 




(0) 



. in p TtTf h»'Si:s ini"i(<.'.e the- r\Vfhvv of pnr ticipints Sf-lcclinri r-prh rospC'^re 



- • (SCniKG COaS: continued) 



^ 14. 


Tlie unit v/ould ho difficult to use for the Durooso 


1 

V 


2 


3 


4 


5 




of trcir.lnti othcru. 


(0) 




a; 








Tho unit Is appropriate for uso in a graduate course 


1 


2 


3 


4 


5 




In educational u('iiuni*iirotion. 


(B) 






(0) 


(0) 


£ 16. 


The unit 1? npproprU^te fo»' use for trmmng pur- 


1 


z 


3 


4 


5 




Doscs in a schoci or district durinci the school 


(0) 


(3) 


1 •/ 




• If / 




year. 












iL. 17, 


I would need r.oro trairiinn in ordfr to serve as 




2 


3 


4 


5 




an Instructor or coordiuatcr with tho unit. 




(5) 




(5) 




18. 


I v;ould bf» vntlinq to use tho unit to train others 




2 


3 


4 


5 




. in I'i'.' *;chotjl or diEtrict. 










(0) 


f 10. 


The unit is too oxpensivo for use in my district. 
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5 




(0) 


a; 










mc unit rccjuiros vcc irucn tirxj to ce usea in o 






3 


4 


5 




school or district during the school year. 


a; 




(3) 







p 21. What specific thinys could b« done to improve the unit and/or the way it was 
! used in the workshop? 



(CornoicntiJ arc ovntainei in E'xfiitit U») 



We would appreciate any additional comments that you wish to make about this unit. 



CC. •.'•./.•*,'/./<■: ,it'c i.ionUvlned in r.xhibit D*) 
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DlffXTtOJiS: Plotise do not write your ntiiiio on this fonn. 

Complete Itctrs 1-20 by circlir.fj n tmhor for each Item to InUtcatc the extent 
to wiiich you atjrtc with tha stiitercnt. Use the following key: 



1 « strong r.riroc'r.^nt with ^tatoirent 
? " ayrecrent with stc»tct cnt 
3 « hoithor ao^ue nor disat/rce 



4 " dicasrec with stiitcnvnt 

5 » sti-ongly ciisagree with i-tatenont 



1. I hiow of other rctorials v;hich cover similar content 
to the i.nit I juit ccrplclcd and which arc nr.ort 
effective for tnntiiny c.oj..inistrQtors . 

2. The unit ir. jottrtially useful for training school 
ac'a.i nisi raters. 

3. It w£'^ difficult to rictcrnine what I v/as supposed 
to Icarn froi;i the unit, 

4. The skills tcu:;ht in the unit are ir;portant for 
school cclii.ir.i St raters to Icsrn. 

5. The instructior"! r.ctlvities 1n the unit provided 
usj^uii ifi r»? Dillon ^nn proctico cn tho ski lls bfing 
ttJii(J:t. 

6. The inotruclioual activities were at about the tifjht 
level of di fficulty. 

7. The SQTTo skills could be learned In far less tiire 
thi'n ti.e unit |:-rei.r:ntly takes. 

8. In general, I liked the instructional activities. 

9. A different fern of practice and assessment would 
have been prefertble to the self- tests In the unit. 

10. I lec'.rnei! a lot from the unit. 

n. I do rot plan to uno either the materials or the 
trainiirg I received in ny schcol or district. 

12. Corprrcd \;ith ether ad?.i1r,i strati ve training I have 
haJ, I would rtuc this unit very low. 

13, 1 wci'ld a('vise other who have responsibility fcr 
ir.ntructir.rc'l prcoriin'. planning to attend a workshop 
bc<:cd cn this unit. 
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ftANAlYZlfIG I^KOBLTO: contlntiod) 



; 14; 


Tho unit v/ould l»o difflcuU to u*:? for the purpose 
of training othurs. 


1 

(ii) 


2 

(0) 


3 

(0) 


4 


5 


1- ■' 


The unit is appiopriato for use in n (jroduiUo cou)*i;e 
In cciucalioncl af^i.iinr.t!'tition. 


1 

(3) 


Z 

(n> 


3 

(3) 


4 

(2) 


5 


£■ . . 


The unit u nf-propnolo for oio for trmninc pur- 
poses in a school or district during the sci;ov)i 
year. 


1 

(PJ 


2 
(9) 


V* 

(S) 


4 


5 
(S) 


1 17. 


I woi'id r.oed iK>rc' triilrn'ro in order to f.or\fi as 
an inutructur cr coordinator v.ith the unit. 


1 

(2) 


Z 
(7) 


3 
(2) 


4 


5 

f5; 


1 18. 


I would be vn'llinq to use tho unit to train oth«»r5 
in 15.,' schcol or district. 


^ 

(2) 


t 

(20) 


3 
(S) 


4 


5 

(2) 


fel9. 

.i." > 


The unit is too expensive for Ui;c in ny district. 

The unit rcc^uires tco ruch tii.x; to be usrtd in a 
school or district curintj tf.e school year. 


1 

(0) 

1 

• (4) 


z 

(B) 

z 

(0) 


3 
(G) 

3 
(Z) 


4 
4 


5 

. 5 
(0) 



?^ 21, What ^ifKcif ic things could be done to improve the unit and/or the way it was 
used in the wurkshop? 

y (C'orj:::::t<: .!;',: r^}.'.ct'.hed in tV.hibic D.) 



}iie would appreciate any additional comments that you wisii to make about this unit. 

(CfjWtCKtc at-:: ocni-ained in Exhibit D.) 



PIKtCTIO:^: Please do not write your ncn.e on this fcrin. 

Complete Itcns 1-20 by circling a nunbor for.cech Item to Indicate the extent 
iti which you agree v.-ith the statcitient. Use the following key: 



1 ' strojuj agrcor.ient with stateiiv^nt 

2 « t!(jre<:r-!Pt with stator.ent 

3 - noithvr a<;roe ncr di i.;if|»*ec 



<l « disagree vMth staten^ont 

5 « stranoly disagree with statcnvint 



^ 1 1 knew of otht r iri.torirls which cover siniiler content 
F ' to'the unit I ju?t cctrpictcd and which are irore 
f- effective for training aOnlnistrators. 

I 2. The unit is potentially useful for training school 

ft aduiinislrators. 

1- 3, It was difficult to dctcrniine what I v/as supposed 

* to learn fron the unit. 

4. ll.p skills tciJf.!;! in t»:o unit are irT^ortant for 

1^. ' schcul acVn.is craters to learn. 

useful infcrnution anc. practice on the skills uting 

■ Kj'.t. 



6, The instructional activities were at about the right 
level of difficulty. 

7, Th:' sv-.o skills cci'ld be learned in far less time 
than the unit presently takes. 

8, In ccncral, I liked the instructional activities. 

9 A different torn of practice and asscssincnt would 
have been profcrcblc to the self-tests in the unit. 

10. I learned a lot from the unit. 

11. I do rot plan to use either the materials or the 
trainTr.Vi* I r('(.e1vcc in ny school or district. 

12 Cr' prr^d \'1th etht.r administrative training I have 
had, fwould rate this unit very lew. 

13 1 vculd advise other v.ho have responsibility for 

' instructicr.M preoran. planning to attend a workshop 
h.^sec. on this unit. 



1 2 3 4 5 

(0) (s) 01) <6) m 



12 3 4 5 

(e) (W (0) (0) (0) 

1 2 3 4 5 

(0) (2) (V (W (V 

1 2 3 4 5, 
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1 2 3 4 5 

(d) (24) (2) (0) (0) 
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1 2 3 4 5, 

(1) (4) (4) (20) (1) 
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(6) (8) (4) (2) (1) 

1 2 3 4 5 

(0) (2) (S) (6) (5) 

1 2 3 4 5 

(0) (3) (V ns; (4) 

1 2 3 4 5 

(S) (8) (S) (3) (1) 
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[ (DERIVING Oa.miVCS: continued) 

1 14, The unit v/ould be diffluuU to Ubo for the purpose 1 2 3 4 5 JJa'^ 
of training others, (B) (^) (4) (6) (i) (h 

I 15, The unit Is eppropriatc for use in a graduate course 

E In educaticriM administration. (S) (14) (0) (0) (0) a 

I?-- ■ 

i: 16, The unit Is oppropriate for use for training pur- 

ftc poses in a sthooi or district during the school (S) (13) (V (3) (0) (t 

p... , yeer» 

17, I would need nore trainirtj In order to serve as 

an instructor or coordintitor with the unit. (2) (9) (4) (4) (1) (1, 



18. I v/ould bo willinfj to use the unit to train others 
• In school or district. (V (iV (V (S) (V (1, 



19. The unit Is too expensive for use In my district. 



20, The urit requires too n-uch tinie to be used In a 

school or district during the school year. (V (B) (6) (4) (S) (2, 
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21. What specific things could be done to Improve the unit and/or the way it was 
used In the workshop? 

( CvPif !v*ita are aontaimd in Eschibit D,} 



We would appreciate any additional comments that you wish to make about thl^ unit. 

(CortTTicnie ave aontaimd in Eschibit D, ) 



(2, 



♦NR = No Response. 
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